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LONG  TERM  GOALS 

The  long-term  goal  of  this  project  is  to  develop  robust  global  and  mesoscale  ensemble  analysis  and 
forecast  systems  that  are  able  to  provide  probabilistic  forecast  guidance  in  an  operational  environment. 
Although  current  guidance  relies  primarily  on  single  deterministic  forecasts  from  numerical  weather 
prediction  models,  recent  advances  in  probabilistic  prediction,  or  ensemble  forecasting,  have  made  this 
technique  a  very  powerful  tool  for  providing  more  complete  input  on  environmental  conditions, 
including  a  measure  of  the  forecast  confidence.  Probabilistic  prediction  of  Navy  relevant  high-impact 
weather  will  significantly  benefit  sea  strike  and  sea  shield  functions  if  there  are  clearly  identified  user- 
relevant  norms.  Ensembles  also  allow  for  the  covariance  between  relevant  weather  variables  to  be 
taken  into  account,  for  example,  indicating  that  if  the  wind  speed  is  high,  then  visibility  will  also  be 
high. 

OBJECTIVES 

The  objective  of  this  project  is  to  incrementally  develop  and  test  an  integrated  multi-scale  ensemble 
modeling  framework  and  forecasting  system,  and  validate  and  transition  it  to  operations  at  the  Fleet 
Numerical  Meteorology  and  Oceanography  Center  (FNMOC).  This  system  will  be  a  state-of-the-art 
global  and  mesoscale  ensemble  forecasting  system  using  the  Navy  Operational  Global  Atmospheric 
Prediction  System  (NOGAPS)  and  the  Coupled/Ocean  Atmosphere  Mesoscale  Prediction  System 
(CO  AMPS®1).  It  will  incorporate  consideration  of  uncertainty  in  the  initial  state  (and  lateral 
boundaries  where  appropriate)  along  with  consideration  of  uncertainty  in  the  model  formulation.  It  will 
be  a  user-friendly,  easily  re-locatable  system  flexible  enough  to  incorporate  new  advances  in  data 
assimilation  and  post-processing  schemes.  The  system  will  provide  high  fidelity,  dynamically 
consistent  probabilistic  forecasts,  and  estimates  of  forecast  uncertainty  for  the  battlespace  environment 
that  will  directly  improve  tactical  decision  aids  (TDA)  and  applications  related  to  homeland  security 
and  the  global  war  on  terrorism. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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APPROACH 


A  global  Ensemble  Transform  (ET)  ensemble  system  has  been  developed  as  part  of  research  performed 
in  recent  NRL  base  projects  on  global  modeling  and  predictability  (Bishop  and  Toth  1999,  McLay  et 
al.  2007).  In  addition,  an  ET  mesoscale  ensemble  system,  driven  by  this  global  ensemble,  has  been 
developed  within  the  FY06-09  NRL  6.2  project  "Probabilistic  Prediction  of  High-Impact  Weather" 
(Bishop  et  al.  2009).  The  outcomes  of  those  projects  have  been  incorporated  into  this  Rapid  Transition 
Process  (RTP)  project,  allowing  us  to  use  the  combined  ONR  and  PMW-120  funding  to  move  rapidly 
forward  with  both  the  global  and  mesoscale  ensemble  research  systems  to  fully  develop,  test,  validate, 
and  operationalize  a  multi-scale  ensemble  forecasting  capability  at  FNMOC.  The  key  individuals 
working  on  this  project  are  Carolyn  Reynolds  and  Justin  McLay  (global  ensembles),  and  Teddy  Holt 
and  Craig  Bishop  (mesoscale  ensembles). 

This  next-generation,  multi-scale  ensemble  system  will  consist  of  (a)  the  representation  of  initial  and 
model  uncertainty  in  NOGAPS,  (b)  the  representation  of  initial  uncertainty,  lateral  boundary  condition 
(LBC)  uncertainty,  and  model  uncertainty  in  COAMPS,  (c)  an  integrated,  robust  global-mesoscale 
ensemble  infrastructure  (to  ensure  consistent  perturbations  between  the  global  and  mesoscale  systems, 
result  in  improvements  to  the  global  system  having  an  immediate  impact  on  the  mesoscale  system,  and 
promote  ease  of  use),  (d)  an  ensemble  verification  system,  including  a  scorecard  for  objectively 
evaluating  overall  skill,  (e)  a  demonstration  of  performance  capabilities  in  an  operational  setting,  and 
(f)  an  ensemble  post-processing  scheme  designed  to  ensure  reliable  probabilistic  forecasts  to  remove 
biases  from  the  forecasts  of  individual  members  together  with  biases  in  the  ensemble  spread.  The 
system  will  be  tested  using  scientific  studies  and  norms  and  comprehensive  ensemble  forecast 
experiments. 

WORK  COMPLETED 

For  the  development  of  the  global  ensemble  system,  a  comparison  between  the  ET  (Bishop  and  Toth 
1999)  and  the  previously  operational  Bred  Vector  (BV,  Toth  and  Kalnay  1993)  scheme  has  shown  the 
ET  scheme  to  be  superior  under  many  different  metrics  (McLay  et  al.  2008).  The  ET  scheme  was 
transitioned  to  FNMOC  in  March  2008.  Research  is  now  underway  to  improve  the  current  fit  of  the 
initial  perturbations  to  the  analysis  error  variance  estimates  produced  by  the  data  assimilation  scheme 
using  a  “banded”  ET  which  should  be  transitioned  to  operations  shortly.  Research  also  investigated 
the  impact  of  including  model  uncertainty  through  inclusion  of  parameter  variations  as  well  as 
incorporation  of  a  simple  model  of  diurnal  SST  variations  (Zheng  and  Beljaars  2005). 

For  the  development  of  the  mesoscale  ensemble  system,  the  ET  method  for  generating  initial 
perturbations  implemented  within  COAMPS  in  2008  has  been  extensively  tested  in  FY09  using  the 
Joint  Mesoscale  Experiment  (JME)  domain  (45-,  15-km;  33  members)  over  Korea,  with  a  focus  on 
developing  and  improving  post-processing  techniques.  Four  bias-correction  techniques  (static, 
Kalman-Filter  (KF),  running-mean,  and  weighted  running  mean)  and  calibration  (ensemble  spread) 
techniques  have  been  developed  and  tested  as  well  as  new  filtered  model  output  statistics  (FMOS)  and 
a  new  approach  to  Bayesian  Model  Averaging  (BMA)  ensemble  post-processing. 
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RESULTS 


a.  Global  ensemble  system:  While  the  first  version  of  the  global  ET  has  already  been  transitioned  to 
operations,  there  are  still  issues  with  the  current  ET  implementation  that  merit  further  research. 
One  issue  is  that  the  ET  scheme  has  too  little  ensemble  variance  in  the  tropics  when  compared  to 
the  analysis  error  variance  estimate  that  is  produced  by  the  operational  data  assimilation  system. 
Recently  a  new  “banded”  ET  in  which  the  transform  is  performed  in  latitudinal  bands  has  been 
developed  to  mitigate  these  problems.  The  new  banded  ET  results  in  a  much  closer  match  to  the 
analysis  error  variance  estimates  and  also  gives  improved  performance  under  a  variety  of  metrics. 
Figure  1  shows  two  examples  of  this  improved  performance,  including  reduced  RMS  error  of  the 
NH  Midlatitude  500-hPa  ensemble  mean  RMS  error,  and  less  redundancy  between  ensemble 
members  at  120-h  indicated  by  higher  values  of  the  normalized  eigenvalues  of  the  ensemble 
covariance  matrix. 


500-hPa  Ensemble-mean  RMS  Error 


Eigenvalues  of  Ensemble 
Covariance  Matrix  at  120  h 


Fig.  1.  Results  for  the  operational  version  of  the  ET  (black  curves)  as  the  new  banded  ET  (red 
curves).  The  left  panel  shows  ensemble  500-hPa  geopotential  height  ensemble-mean  forecast  error 
for  20-65N,  as  a  function  of forecast  time.  The  right  panel  shows  the  normalized  eigenvalues  of  the 

ensemble  covariance  matrix  at  120  h. 


Another  way  to  improve  the  current  ET  implementation  is  through  inclusion  of  model  uncertainty. 
Experiments  combining  the  ET  with  parameter  variations  in  just  the  convective  scheme  are  compared 
with  parameter  variations  in  both  the  convective  and  boundary  layer  scheme.  It  is  found  that  while 
parameter  variations  in  the  convective  scheme  improve  performance  in  the  tropics,  they  have  little 
impact  on  mid-latitudes.  In  contrast,  variations  in  both  the  convective  and  boundary  layer  parameters 
provide  improved  performance  in  both  the  tropics  and  the  mid-latitudes.  Figure  2  shows  improved 
performance  in  the  ability  of  the  ensemble  variance  to  differentiate  between  a  wider  range  of  ensemble 
mean  forecast  errors  when  both  the  convective  and  boundary  layer  parameters  are  varied  (green  curve) 
as  opposed  to  just  varying  the  convective  parameters  (cyan  curve)  or  the  control  (dark  blue  curve). 
Experiments  are  also  being  carried  out  with  the  addition  of  a  simple  model  of  diurnal  SST  variations 
following  Zheng  and  Beljaars  (2005).  Experiments  show  mostly  neutral  to  improved  performance  in 
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standard  ensemble  metrics  (e.g.,  improved  relationship  between  ensemble  variance  and  forecast  error 
variance,  cyan  curve  in  Figure  2),  as  well  as  improved  simulation  of  equatorial  waves  such  as  Kelvin 
waves  (not  shown),  which  should  improve  prediction  of  the  Madden  Julian  Oscillation. 


Fig.  2.  Ensemble  mean  forecast  error  variance  compared  to  ensemble  variance  for  NH  midlatitude 
500-hPa  height  48-h  forecasts,  May-June  2007.  Dark  blue:  Control  ensemble  (no  model 
uncertainty).  Magenta:  Variations  in  convective  parameters.  Cyan:  Diurnal  SST  variations. 

Green:  Variations  in  convective  and  boundary  layer  parameters. 

b.  Mesoscale  ensemble  system:  New  post-processing  techniques  have  been  developed  and  tested 
using  the  JME  domain  for  the  3-month  time  period  (12  February  to  15  May  2009)  using  340  fixed 
surface  observation  stations  for  air- temperature,  wind  speed,  and  dew  point  depression.  Figure  3 
shows  an  example  of  the  improvement  that  KF  bias  correction  affords  over  no  bias  correction  (raw) 
for  2-m  air  temperature.  Generally,  weighted  running  mean  and  KF  bias  correction  perform  better 
than  either  running  mean  or  static  bias  correction. 
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Fig.  3.  CO  AMPS  mean  48-h  forecast  bias  of  2-m  air  temperature  (K)  for  the  observations  in  the  15- 
km  JME  domain  for  15  April  to  15  May  2009 for  a)  raw  (no  correction),  and 

b)  Kalman-Filter  correction. 

In  addition  to  bias  correction,  calibration  of  second  moments  (spread)  have  also  been  developed,  with 
the  second  moment  estimated  as  a  linear  function  of  the  ensemble  sample  covariance.  Figure  4  shows 
the  calibrated  Taligrand  diagram  and  reliability  for  the  KF-calibrated  48-h  forecasts  of  2-m  air 
temperature.  The  probability  integral  diagram  (Fig.  4a)  shows  an  approximately  flat  distribution  as 
desired,  in  contrast  to  an  under-dispersive  non-calibrated  distribution  (Figure  not  shown).  Likewise 
the  reliability  diagram  (Fig.  4b)  for  the  median  air  temperature  (blue  line),  10  percentile  (red),  30 
percentile  (green),  70  percentile  (cyan)  and  90  percentile  (magenta)  all  show  excellent  agreement  of 
forecast  probability  with  relative  frequency  across  all  ranges. 
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Fig.  4.  COAMPS  15-km  JME  48-h  KF-calibrated  2-m  air  temperature  (K)  forecasts  from  12 
February  to  9  June  2009 for  a)  Probability  integral  transform  ( PIT)  histogram,  and  b)  reliability. 


The  newly  developed  weather  forecasting  tool  (FMOS)  improves  existing  statistical  corrections  of 
systematic  forecast  error,  particularly  when  the  training  data  set  is  small  (Bishop  2009,  US  provisional 
patent  68/181686).  Previous  MOS  systems  (i.e.,  Wilson  and  Vallee  2002)  use  blended  regression 
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equations  based  on  earlier  versions  of  the  model  with  equations  based  on  more  recent  versions  of  the 
model,  assuming  that  significant  model  changes  are  typically  separated  by  more  than  two  years.  This 
assumption  is  not  satisfied  for  US  Navy  operations  in  which  model  domains  are  typically  deployed  in  a 
specific  theatre  for  a  period  between  two  weeks  and  two  years.  In  principal,  new  MOS  equations  need 
to  be  developed  for  each  unique  theatre  in  which  the  forecasting  model  is  deployed.  Hence,  the  Navy’s 
deployment  periods  are  too  short  for  the  estimated  regression  coefficients  to  stabilize.  Any  attempt  to 
estimate  the  regression  coefficients  using  MOS  with  such  a  small  data  set  will  result  in  very  noisy 
regression  coefficients.  The  new  FMOS  developed  here  stabilizes  and  spatially  smoothes  such  noisy 
regression  coefficients  associated  with  small  training  data  sets  by  optimally  combining  a  data-poor 
MOS  estimate  with  a  smooth  prior  estimate.  MOS  methods  assume  that  on  the  kth  day/event,  the  ith 
model  forecast  variable  fk  is  related  to  the  corresponding  verifying  analysis/observation  variable  yf 

via  the  stochastic  process  f]k  =  atyk  +  bi  +  s\  where  s\  is  a  random  number  with  mean  zero  and 
variance  a]  that  is  statistically  independent  of  yk.  .  Fig.  5  a  gives  an  example  of  the  estimates  afos  of 
aj  one  would  obtain  using  the  standard  MOS  equations  and  simulated  meteorological  data  when  just 

16  consecutive  meteorological  events  were  used  to  make  the  estimate.  Fig.  5c  gives  the  corresponding 
true  values.  The  beneficial  effect  of  the  FMOS  tool  is  evident  in  the  comparison  of  Fig.  5a  and  5e 
indicating  that  while  the  MOS  estimate  of  the  regression  coefficient  is  noisy,  the  FMOS  estimate  is 
smooth.  Comparison  of  the  contour  scales  for  Fig.  5b  and  5f  shows  that  the  maximum  errors  incurred 
by  the  MOS  estimate  are  more  than  5  times  larger  than  those  incurred  by  the  FMOS  estimate,  and  the 
mean  square  error  (mse)  of  the  MOS  estimate  is  21  times  larger  than  the  mse  of  the  FMOS  estimate. 
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Fig.  5:  Comparison  of  estimates  of  the  slope  coefficient  a  from  MOS  and  FMOS  from  16 
realizations.  In  (a)  and  (b)  are  shown  the  MOS  estimate  of  a  and  its  error,  respectively.  The  true  a- 
field  is  shown  in  (c)  while  (d)  indicates  the  spatial  covariance  function  of  the  a-field  obtained  from  a 
spectral  analysis  of  the  MOS  estimate  shown  in  (a).  The  FMOS  estimate  of  a  and  its  error  are 

shown  in  (e)  and  (f),  respectively. 


The  advantages  of  FMOS  over  MOS  are  that  (a)  its  estimates  of  regression  coefficients  are  more 
accurate  -  particularly  when  the  training  data  set  is  small,  and  (b)  its  estimates  of  regression 
coefficients  lie  between  those  of  a  perfect  forecast  model  and  those  of  the  actual  model.  It  can  be 
shown  that,  if  properly  configured,  FMOS  will  deliver  equivalent  results  to  MOS  in  the  limit  of  an 
infinite  data  training  set. 

Finally,  BMA  methods  of  ensemble  post-processing  in  which  continuous  probability  density  functions 
(pdfs)  are  constructed  from  ensemble  forecasts  by  centering  functions  around  each  of  the  ensemble 
members  have  been  examined.  The  primary  purpose  of  BMA  methods  is  to  correct  systematic 
mismatches  between  ensemble  variance  and  the  error  variance  of  the  ensemble  mean.  In  Bishop  and 
Shanley  (2008),  idealized  ensemble  forecasting  experiments  were  used  to  show  that  these  methods  are 
liable  to  produce  systematically  unreliable  probability  forecasts  of  climatologically  extreme  weather. 
They  showed  that  the  failure  of  these  methods  is  linked  to  an  assumption  that  the  distribution  of  truth 
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given  the  forecast  can  be  sampled  by  adding  stochastic  perturbations  to  state  estimates,  even  when 
these  state  estimates  have  a  realistic  climate.  The  research  demonstrates  that  this  assumption  is 
incorrect  and  argues  that  such  dressing  techniques  better  describe  the  likelihood  distribution  of 
historical  ensemble  mean  forecasts  given  the  truth  for  certain  values  of  the  truth.  This  paradigm  shift 
led  to  a  new  approach  that  incorporates  prior  climatological  information  into  BMA  ensemble  post¬ 
processing  via  Bayes’  theorem.  This  new  approach  is  shown  to  cure  BMA’s  ill  treatment  of  extreme 
weather  by  providing  a  posterior  BMA  distribution  whose  probabilistic  forecasts  are  reliable  for  both 
extreme  and  non-extreme  weather  forecasts.  Work  is  underway  to  incorporate  this  new  ensemble  post¬ 
processing  technique  into  the  coupled  model  ensemble. 

IMPACT/APPLICATIONS 

Since  results  indicate  improved  performance  of  the  global  ET  system  over  the  previous  operational 
method  under  a  variety  of  metrics,  this  new  scheme  is  resulting  in  more  skillful  global  ensemble 
forecasts  for  the  Navy.  This  approach  should  have  a  direct  impact  on  current  and  planned  ensemble- 
based  products,  such  as  the  probability  of  gale-force  winds,  as  well  as  other  products  that  are  currently 
forced  by  the  global  ensemble,  or  will  be  in  the  future,  including  the  mesoscale  atmospheric 
ensembles,  and  surface  ocean  wave  ensembles.  In  the  future,  it  is  hoped  that  these  more  skillful 
probabilistic  tools  will  lead  to  more  applications  of  probabilistic  forecasts  for  the  Department  of 
Defense,  including  ship  and  air  routing  applications. 

TRANSITIONS 

Based  on  the  superior  performance  of  the  ET  as  compared  to  the  previously-operational  bred-vector 
scheme,  the  ET  scheme  was  transitioned  to  the  operational  global  ensemble  system  in  March  2008.  It 
is  anticipated  that  the  banded  ET  will  be  transitioned  to  operations  in  the  next  few  months,  and 
inclusion  of  model  uncertainty  transitioned  to  operations  in  FY 10.  In  addition,  as  part  of  the  Navy/Air 
Force  Joint  Mesoscale  Ensemble  (JME)  forecasting  project  we  have  aided  FNMOC  in  the 
implementation  and  execution  of  a  “beta-ops”  real-time  mesoscale  ensemble  forecasting  system  over 
the  Korean  peninsula.  The  COAMPS  ET  ensemble  has  been  running  continuously  on  the  High 
Performance  Computing  network’s  DC3  system  since  late  June  2008.  The  transition  to  operations  is  in 
progress,  with  porting  to  the  FNMOC  large  operational  LINUX  cluster  (OPAL  A2)  complete.  The 
mesoscale  system  has  been  successfully  tested  by  NRL  in  a  quasi-operational  configuration  at 
FNMOC  using  the  operational  data  streams. 

RELATED  PROJECTS 

The  global  ET  and  Stochastic  Physics  research  began  in  preceding  years  under  the  NRL  base-funded 
project  6.1 -Quantifying  Limits  of  Atmospheric  Predictability  and  the  NOAA  funded  project  6.2 
Stochastic  Physics.  In  addition,  related  research  has  been  done  under  the  NRL  base-funded  project  6.2- 
Model  Discovery,  which  uses  ensembles  to  understand  model  behavior  within  parameter  space,  and 
the  NOAA-funded  Hurricane  Forecast  Improvement  Project  (HFIP).  For  the  mesoscale  ensemble 
development  related  projects  include  ONR  funded  (through  the  NRL  Base  program)  6.2-Probabilistic 
Forecasting  of  High  Impact  Weather,  6.2-Hidden  Volatility  in  Environmental  State  Estimation,  and 
6.1 -Quantifying  Limits  of  Atmospheric  Predictability.  The  research  is  also  related  to  the  NOAA 
funded  projects  6.2  Stochastic  Physics  and  6.2  Huge  Ensemble  State  Estimation. 
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